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This study evaluates the quality of cost estimates 
produced by each of four cost progress models--a random walk 
model, the traditional learning curve model, a production rate 
model (fixed-variable model), and a model incorporating both 
learning curve and production rate effects (Bemis production 
rate adjustment model) . Emphasis is on assessing the level of 
bias associated with these models and determining the 
influence of various factors on model performance. Findings 
indicate, on average, the learning curve and Bemis models 
underestimate unit costs, while the random walk and fixed- 
variable models overestimate unit costs. Different factors are 
evaluated to determine their significance in explaining 
variations in the bias of unit cost predictions and 
relationships between the significant variables and model cost 
prediction bias are described. Findings indicate the Bemis 
model is superior to the other cost progress models because it 
exhibits the least bias and is not significantly influenced 
(in terms of bias) by variations in the factors considered. 



iii 



//A?w 

//& 

Cj 



TABLE OF CONTENTS 



I. INTRODUCTION 1 

A. DEFENSE SPENDING AND PUBLIC INTEREST 1 

1. The Federal Budget Deficit 1 

2. Cost Overruns in Weapons Acquisition .... 2 

B. COST ESTIMATION MODELS 4 

1. The Learning Curve 5 

2. Alternative Cost Progress Models 6 

C. PURPOSE 6 

D. RESEARCH QUESTIONS 7 

E. SCOPE, LIMITATIONS AND ASSUMPTIONS 7 

F. ORGANIZATION OF THESIS 8 

II. REVIEW OF PREVIOUS RESEARCH 9 

A. SURVEY OF RESEARCH PRIOR TO 197 0 10 

1. Hirsch 10 

2. Cochran 10 

3. Alchian 12 

B. SURVEY OF RESEARCH DURING THE SEVENTIES .... 13 

1. Linder and Wilbourn 13 

2. RAND Studies 16 



a. Large, Hoffmayer and Kontrovich .... 16 

b. Large, Campbell and Cates 17 



IV 



3. Smith 18 

C. SURVEY OF RESEARCH DURING THE EIGHTIES .... 19 

1. Crouch 19 

2. Smith 2 0 

3. Balut 21 

4. Bemis 23 

D. SURVEY OF RESEARCH DURING THE NINETIES .... 24 

1. Boger and Liao 24 

2. Moses 27 

E. SUMMARY 29 

III. SAMPLE, DATA AND VARIABLES 3 2 

A. SELECTION OF THE SAMPLE 3 2 

1. Data Sources 32 

2. Selection Process and Criteria 34 

3. Programs Selected for Study 34 

B. COMPILATION OF PROGRAM DATA 3 7 

C. EXPANSION OF THE SAMPLE 3 8 

D. SELECTION OF COST PROGRESS MODELS 3 9 

1. Random Walk Model 4 0 

2. Traditional Learning Curve Model 41 

3. Fixed-Variable Model 42 

4. Bemis Production Rate Adjustment Model ... 43 

E. ANALYSIS OF BIAS 4 4 

1. Unit Cost Prediction 44 

2. Bias Measurement-The Dependant Variable . . 46 



v 



3. Explanatory Variable Selection 47 

a. Demographic Variables 49 

b. Condition Variables 49 

(1) Burden 50 

(2) Cost Variance 51 

(3) Learning Rate 52 

(4) Production Rate Variance 53 

(5) Future Production Level 54 

(6) Past Production Trend 55 

(7) Plot Points 56 

F. SUMMARY 57 

IV. ANALYSIS AND FINDINGS 58 

A. STATISTICAL PROCEDURES 58 

B. ANOVA RESULTS 60 

C. OVERALL MODEL PERFORMANCE 62 

1. Mean Bias 62 

2 . Median Bias 64 

3 . Model Bias Correlations 65 

D. REGRESSION ANALYSES AND RESULTS 67 

1. Random Walk Model Bias (BIASRW) 68 

a. LRATE 7 3 

b. FUTUPROD 7 3 

2. Learning Curve Model Bias (BIASLC) 74 

a. BURDEN 7 6 

b. LRATE 79 



vi 



c. BEGTREND SO 

d. ENDTREND 3 3 

e. FUTUPROD 3 6 

f. MOD 37 

3. Fixed-Variable Model Bias (BIASFV) 89 

4. Bemis Rate Adjustment Model Bias (BIASBE) . 92 

E. SUMMARY 94 

V. SUMMARY 9 6 

A. FINDINGS 96 

B. FUTURE RESEARCH OPPORTUNITIES 100 

LIST OF REFERENCES 101 

INITIAL DISTRIBUTION LIST 103 



I. INTRODUCTION 



A. DEFENSE SPENDING AND PUBLIC INTEREST 

In recent years, spending for national defense has 
increasingly been the focal point of public scrutiny. This 
trend can be attributed to two underlying causes — the growing 
federal budget deficit, and the problem of cost overruns in 
the acquisition of major weapons systems. 

1. The Federal Budget Deficit 

Between fiscal years 1980 and 1987, the 
Department of Defense annual Budget Authority 
almost doubled, from $143 billion to $281 
billion, with its total exceeding $1 trillion 
during that period. This sharp increase 
contributed to rising deficits and aroused 
public concern over the ways defense dollars 
were being spent. [Ref. l:p. 8] 

The Congressional Budget Office's "Economic and Budget 
Outlook: Fiscal Years 1992-1996" indicates that since 1986, 
the defense budget has been on a downward path. Total budget 
authority 1 for 1991 was down approximately four percent from 
1990. Moreover, projections for 1992 and 1993 indicate 
further decreases. [Ref. 3:p. 84] As a result of these 



Budget authority is "the authority granted to a federal 
agency in an appropriations bill, to enter into commitments that 
result in immediate or future spending." Budget authority is not 
necessarily the amount of money that an agency or department will 
spend during a fiscal year. Instead, it is merely the upper limit 
on the amount of new spending commitments it can make. [Ref. 2:p. 
176] 
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trends, it is more important than ever that the Department of 
Defense (DoD) manages its dollar resources effectively. 

One key area in which costs must be managed 
effectively is the development and procurement of weapons 
systems 2 . 

During much of the past three decades, constant dollar 
unit costs for major defense systems have grown much 
faster than constant dollar total budgets for these 
systems. The result has been the purchase of smaller 
quantities of new systems, delayed modernization and 
shrinking capabilities. [Ref. l:p. 10] 

Budget authority and outlays for research, development, test 

and evaluation and procurement accounted for approximately 

forty percent of the national defense budget in 1989 [Ref. 

4:p. A-146]. More effective and efficient utilization of 

these funds could result in significant savings within the 

DoD, thus allowing scarce resources to be applied to other 

important requirements. 

2. Cost Overruns in Weapons Acquisition 

The second underlying cause of increased public 

scrutiny of defense spending is cost overruns. The problem of 

cost overruns in the weapons acquisition process has been a 

major source of consternation and embarrassment for the DoD 

for many years. Numerous researchers and presidential 



2 The term weapon system normally refers to the major item of 
equipment and the subsystems, logistical support, software, 
construction and training needed to operate and support it [Ref. 
l:p. 9]. For purposes of this thesis, weapon system refers only to 
the major item of equipment. 
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commissions during the past thirty years have concluded that 
tens of billions of dollars per year could be saved by 
improving the acquisition process [Ref. l:p. 32]. "The 
studies (have) repeatedly urged Congress and the Defense 
Department to correct five basic deficiencies: 



1. Setting requirements for the most sophisticated 
systems available, often irrespective of cost; 

2. Changes in program and contract requirements caused by 
changes in military user preferences, leading to 
annual or more frequent changes in program funding 
levels, initiated by Congress and DoD itself; 

3 . Lack of incentives for contractors and government 
personnel to reduce program costs; 

4. Failure to develop sufficient numbers of military and 
civilian personnel with training and experience in 
business management and in dealing with industrial 
firms to oversee the development and production of 
enormous, highly technical industrial programs; and 

5. Underestimated schedules and costs of major programs, 
distorting the decision-making process for the 
allocation of the national budget." [Ref. l:p. 32] 



While progress has been made in each of the areas 
identified above, there remains much room for improvement. 
Major defense procurement programs have repeatedly experienced 
significant unanticipated schedule delays and cost increases. 
More than ninety percent of these programs exceed initial cost 
estimates and, in the majority of cases, the average increase 
in cost has been more than fifty percent, excluding the 
effects of quantity changes and inflation. [Ref. l:p. 32-33] 
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A cost overrun occurs when the actual cost of a 



program exceeds the estimated cost. Cost overruns typically 
occur when: fair initial cost estimates are made but 
subsequent actual costs are poorly managed and controlled; or 
actual costs are well-managed but initial cost estimates were 
unrealistic. [Ref. 5:p. 1] 

There are various reasons why initial cost estimates 
may be unrealistic, particularly unrealistically low. For 
example, institutional incentives may exist both within DoD 
and at government contractors to underestimate costs initially 
in order to get a program approved and started. A second 
possible reason is that processes, techniques or tools for 
creating cost estimates may be weak and provide misleading 
cost forecasts. This thesis addresses a technical question 
related to the latter issue. In particular, this paper 
examines the problem of low/unrealistic cost estimates for 
major weapons programs by analyzing the performance of 
alternative cost estimation models. 

B. COST ESTIMATION MODELS 

Broadly, cost estimation models fall into two categories. 
First, Cost Estimating Relationships (CERs) attempt to explain 
or predict the cost of a "standard" unit of an item to be 
manufactured or procured in terms of variables reflecting 
qualities, attributes or characteristics of the item. For 
example, aircraft costs may be modeled in terms of speed or 
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payload. Second, Cost. Progress Models attempt to explain or 
predict changes in unit cost of items over the life of a 
production or procurement program in terms of changes in the 
circumstances surrounding production or acquisition. For 
example, costs may be expected to depend on the numbers 
acquired and the production rate and thus unit costs may be 
modeled in terms of such variables. The analysis in this 
thesis will focus on the latter type of model, cost progress 
models . 

1. The Learning Curve 

The most commonly used cost progress model is the 
learning curve. 

Learning curves have gained widespread acceptance as a 
tool for planning, analyzing, explaining and predicting 
the behavior of the unit cost of items produced from a 
repetitive production process [Ref. 5:p. 1]. 

Although learning curves were originally developed and applied 
to predict cost and time requirements for the construction of 
ships and aircraft during World War II, they have since been 
applied in many other manufacturing and non-manufacturing 
settings. The learning curve phenomena was first reported by 
T. P. Wright in the Journal of Aeronautical Sciences in 1936. 
Wright observed that, as the quantity of units manufactured 
doubles, the number of direct labor hours/cost associated with 
the production of an individual unit decreases at a uniform 
rate. Moreover, the uniform rate of learning is peculiar to 
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the manufacturing process being observed. 

2 . Alternative Cost Progress Models 

It is generally acknowledged that other factors, in 
addition to cumulative quantity, influence unit cost and that 
the simple learning curve does not provide a fully adequate 
description of cost behavior. As a result, prior research has 
attempted to improve the simple learning curve model by 
including additional variables. [Ref. 5:p. 2] 

There are now multiple approaches and models available 
for estimating the costs of acquisition programs. Two of the 
most commonly used cost progress model types are the learning 
curve and the production rate adjustment model. Other model 
types include the plateau model, the Stanford-B model, the De 
Jong model and the S-model. These models are differentiated 
by the variables included and the underlying assumptions. 

C. PURPOSE 

It is unclear at present which model type is most 
appropriate for predicting costs under various manufacturing 
conditions. The purpose of this thesis is to evaluate the 
quality of cost estimates produced by each of four cost 
progress models — a random walk model, a learning curve model, 
a production rate model, and a model incorporating both 
learning curve and production rate effects. In conducting 
this evaluation, emphasis will be place on assessing the level 
of bias associated with each of these models. 
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D. RESEARCH QUESTIONS 

This thesis will address the following research questions: 

1 . Primary Research Question 

What is the bias exhibited by available cost progress 
models when predicting the future unit cost of weapons systems 
acquired through a continuing acquisition program? 

2 . Subsidiary Research Questions 



a. Are the various available cost progress models 
comparable in terms of bias? 

b. Do particular models result in less biased 
estimates under certain conditions? 

c. What are those conditions that affect the 
performance of the models? 

d. Can guidelines be established for determining when 
(under what conditions or circumstances) it is 

most appropriate to use a particular model type? 



E. SCOPE, LIMITATIONS AND ASSUMPTIONS 

There are a number of different criteria which can be 
examined in order to assess various aspects of model 
performance. Two of these criteria which are particularly 
important are the accuracy and the level of bias associated 
with a particular model. Accuracy refers to the degree of 
error in a model's prediction, without regard to the direction 
of the error. Bias refers to both the direction and the 
magnitude of error. It indicates whether predictions made 
using a particular model underestimate or overestimate actual 
cost. The focus of this thesis will be limited to an analysis 
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of the bias associated with the various models tested. One 



purpose of the study is to either confirm or disconfirm the 
results of an earlier simulation study by Moses ( Learning 
Curve and Rate Adjustment Models: An Investigation of Bias ) 
and to determine whether or not those results hold when 
testing real world data. A second purpose is to extend the 
analysis of bias to a larger set of models. 

F. ORGANIZATION OF THESIS 

The remainder of this thesis is organized into four 
chapters. Chapter II provides a review of the literature 
dealing with various cost progress models. Chapter III 
provides a description of the sample, data and measures used 
to conduct the study. Chapter IV provides an analysis of the 
results. Finally, chapter V summarizes the research findings 
and suggests directions for future research. 
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II. REVIEW OF PREVIOUS RESEARCH 



Although the progress or learning curve technique of cost 
estimation was discovered prior to World War II, several 
decades passed before statistical studies of this phenomena 
could be readily conducted. This situation can be attributed 
to two underlying problems: (1) the available data were 
frequently too sparse to support statistical analyses, and; 
(2) the sheer volume of calculations and lack of powerful 
computers meant that many laborious hours were required to 
perform operations which can today be performed in a few 
minutes. This second problem had a strong inhibiting effect 
on researchers. As a result, it was not until the 1950's that 
significant statistical studies of the learning curve 
phenomena began to be undertaken. [Ref. 6:p. 8] 

This chapter reviews the literature dealing with 
development of the learning curve and alternative rate 
adjustment models. In particular, it summarizes the findings 
of some of the major studies which have been conducted in an 
effort to evaluate the relative performance of these models. 
The discussion is organized chronologically into the following 
groupings: research prior to 1970, research during the 
seventies, research during the eighties, and research during 
the nineties. 
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A. SURVEY OF RESEARCH PRIOR TO 1970 

1. Hirsch 

In 1952, Hirsch [Ref. 7] published the results of a 
five-year study of a large United States machine builder. The 
purpose of the study was to examine the relationship between 
labor requirements and production volume. Lot size was used 
as a measure of the rate of production based on the existence 
of stable production lot intervals. In addition, practically 
no changes in management or plant and equipment occurred 
during the period of study. Based on the results of his 
study, Hirsch concluded the relation between direct labor 
requirements and lot size was of little consequence in the 
machining and assembling processes. 

2. Cochran 

In 1960, E. B. Cochran [Ref. 8] published an article 
in which he conducted a careful examination of the basic cost 
function — the learning curve--by studying specific 
manufacturing conditions and parameters which relate them to 
cost trends. The purpose of this study was to probe various 
learning curve applications in an effort to develop new 
concepts of learning curve analysis and revised applications. 
Based on the results of his study, Cochran proposed 
refinements to the basic learning curve concept. In 
particular, he suggested that learning may not necessarily 
occur exactly once per unit but instead, may occur either 



10 



faster or slower. For example, when planning airframe costs 
for a four-engine aircraft, it may be appropriate to consider 
each engine pod as a unit of learning rather than simply 
consider the aircraft as a unit of learning. 

Cochran provided guidelines for accurately identifying 
the unit of learning and identified other factors, in addition 
to worker learning, that would affect the rate of cost 
reduction (e.g., tooling, supervision, parts design and 
shortages) . Moreover, he demonstrated how changing rates of 
learning and task changes may result in both shifts of the 
learning curve and non-linear learning. One of the most 
significant findings presented by Cochran was that: 

Any change in learning rate is equivalent to a change in 
the unit at which standard cost is reached. And this in 
turn generates a major shift in the entire level of cost 
[Ref. 8 : p . 319]. 

As a result of this finding, Cochran concluded "that 
the determination of learning rate is of major significance in 
forecasting and controlling costs." [Ref. 8:p. 319] 

Cochran indicated that the shape of the learning curve 
can be critical in the first 100 units and pointed out the 
fact that there is a wide range of error in straight line 
curves. Accordingly, he suggested that an S-curve pattern may 
be more appropriate than the usual linear learning curve. 
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3. Alchian 



In 1963, Alchian [Ref. 9] published the results of a 
1949 study conducted for the RAND Corporation. The purpose of 
this study was to examine the similarity of airframe 
manufacturing progress functions among various airframe 
manufacturers. Statistical tests of the similarity of the 
functions among various airframe manufacturers were performed 
using World War II data. In addition, the reliability of 
predictions made with these curves was assessed. 

The results of this study indicated the progress 
functions differed among various airframe types and 
manufacturing facilities both in the amount and rate of change 
of required direct labor per pound of airframe. Alchian 
suggested that, for practical purposes, the use of an average 
of individual progress functions may be appropriate. By 
applying this procedure to 22 airframes produced at different 
facilities, the average production error 3 was found to be 



Production error was defined as follows: 

Production Error = Mediated Manhours - Ac tual Manhours 

Actual Manhours 

Direct labor requirements (manhours) for the first 1000 planes were 
predicted for 22 aircraft model — facility combinations using both 
an industry progress curve and an airframe type progress curve. 
The percentage error resulting from the use of each of these curves 
was then computed for each model — facility combination using the 
equation described above. Next, the weighted average error per 
facility (weighted by actual manhours) associated with the use of 
the industry and airframe progress curves was computed for each of 
the four major aircraft model groups examined: bombers, fighters, 

trainers and transports. Based on these figures, the weighted 
average error per facility for all facilities was computed. 
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approximately 25 percent. This same result was obtained for 
che entire output of any particular airframe produced in one 
facility. Specific curves fitted to the past performance of 
a particular manufacturing facility resulted in margins of 
error of approximately 20 percent. 

Alchian examined alternative relationships between 
direct labor per pound of airframe, cumulative number of 
airframes, time and rate of production. "The results cast 
doubts on any of the alternatives being better fits than the 
usual progress curve." [Ref. 9:p. 692] 

B. SURVEY OF RESEARCH DURING THE SEVENTIES 
1. Linder and Wilbourn 

Models which relate costs to various cost-driving 
features or parameters (i.e., physical/performance parameters 
of the weapons system) typically result in reasonable 
estimates of future recurring unit procurement costs [Ref. 
10:p. 277]. Nevertheless, Linder and Wilbourn [Ref. 10] 
suggested that in addition to these parameters, various 
characteristics of the procurement program itself (e.g., 
competitive versus sole-source procurement, single-year versus 
multiple-year buys, and low versus high production rates) 
represent cost-driving features which should be accounted for 
in the cost estimating procedure. 
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Linder and Wilbourn investigated the effect of 
production rate on recurring missile unit procurement costs. 
In particular, they developed two models to examine how 
production rate influences the position and/or slope of the 
recurring missile hardware cost improvement curve. The first 
model formulated unit recurring cost as a function of a 
constant "annual" production rate. The second model 
formulated unit recurring cost as a function of a variable 
annualized production rate. 

These models were developed based on an analysis of 
the impact of production rate changes on direct and indirect 
costs. Linder and Wilbourn reasoned that higher production 
rates would result in lower fixed costs per unit. Moreover, 
high production rates were expected to lead to a smaller 
percentage increase in indirect costs than direct costs. As 
a result, lower overhead rates should be applied to direct 
costs. Based on the combined effects of lower fixed costs per 
unit and lower applied overhead rates, Linder and Wilbourn 
concluded that unit costs would be reduced at higher 
production rates, ceteris paribus. Moreover, the cost 
improvement curve associated with a high production rate was 
expected to lie below a cost improvement curve associated with 
a lower production rate. 

In assessing the expected effect of production rate on 
the cost improvement curve slope, the researchers made the 
following observations: 
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1. At high production rates, average fixed direct costs are 
reduced and variable direct costs constitute a larger 
portion of total direct costs. 

2. As direct costs fall, indirect costs are reduced at a 
slower rate such that overhead rates increase. 

These two phenomena have opposite affects on the slope of 
the cost improvement curve. The first one will tend to 

increase the slope of the curve due to the influences of 
learning and other related effects on variable directs costs 
at a high production rate. The latter phenomena will tend to 
flatten the curve as production increases. As a result, "the 
net effect on the cost improvement curve slope depends on the 
relative amounts of direct and indirect costs per unit as well 
as the proportion of each cost category which can be 
considered as fixed or variable." [Ref. 10:p. 280] 

Based on the results of their analyses, Linder and 
Wilbourn reached the following conclusions: 

1. Ceteris paribus, higher production rates result in lower 
unit recurring costs at each production quantity. 

2. Doubling the production rate lowers average unit costs by 
approximately three to seven percent for the quantities 
examined . 

3. Changing the production rate has only a slight influence 
on the slope of the unit cost improvement curve. 

4. The effects discussed above are relatively insensitive to 
changes in the models' parameter values. [Ref. 10 :p. 300] 
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RAND Studies 



2 . 

During the 1970's, the RAND Corporation conducted a 
number of studies which examined the relationship between 
production rate and airframe costs. The specific objectives 
and results of two of these studies are summarized below, 
a. Large, Hoffmayer and Kontrovich 

In 1974, the results of a study by Large, Hoffmayer 
and Kontrovich [Ref. 11] were published. "The purpose of this 
study was to investigate the nature, magnitude and causes of 
the influence of production rate on unit cost." [Ref. ll:p. 
iii] Based on the assumption that production rate and unit 
cost vary inversely, the researchers sought to develop an 
estimating model to express the relationship for various 
elements of cost. 

The results of the analysis suggested the effect of 
production rate on manufacturing labor, manufacturing 
materials, tools and engineering could not be predicted with 
confidence. In any specific case, the effect depended on a 
number of factors including how rate changes were achieved, 
the availability of suppliers, the local labor supply, 
management policy, the timing of rate changes, plant capacity, 
and plant backlog. The only element of cost which was found 
to clearly be a function of production rate was overhead. 

Based on their findings, the researchers concluded 
that the influence of production rate on aircraft cost could 
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not be predicted with any degree of confidence. Each case 
should be examined separately and in detail to assess the 
effect of rate. In addition, they suggested that in advanced 
planning studies, rate effects in aircraft production programs 
can be ignored because they are far outweighed by other 
uncertainties. As a result, they indicated that a model that 
does not explicitly consider rate may be preferable for 
advance planning purposes. 

b. Large, Campbell and Cates 

This study, published in 1976, attempted to derive 
improved parametric equations for estimating the acquisition 
cost of aircraft airframes. Earlier RAND studies had 
indicated variations in cost among different airframes were 
best explained by the quantity produced and aircraft 
characteristics, i.e., airframe unit weight and maximum speed. 
Large, Campbell and Cates [Ref. 12] were unable to identify 
additional characteristics that would make an estimating model 
more flexible and, hence, better able to deal with 
characteristics peculiar to individual aircraft. None of the 
independent variables considered significantly improved the 
reliability of estimates obtained using only weight, speed and 
cumulative quantity. As a result, they suggested that future 
research which examines the influence of program 
characteristics on program cost may be more productive. 
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Smith 



3 . 

The purpose of this study was to develop and test a 
procedure to assess the effect of production rate changes on 
the direct labor requirements for production of additional 
airframes. Smith [Ref. 13] proposed a cost model to express 
direct labor hours required as a function of cumulative 
production and production rate. Two approaches to expressing 
the production rate variable were examined: a lot average 
delivery rate, and a lot average manufacturing rate. Data 
from three airframe production programs — the F-4 program, the 
F— 102 program, and the KC-135 program — were used to construct 
data sets. These data sets were then examined in the 
cumulative production and production rate cost model using 
regression analysis. 

Production rate was found to be an important factor in 
the cost of airframe production, although its effect was 
subordinate to that of cumulative production. The study 
demonstrated empirically that production rate can be an 
important predictor of variation in unit direct labor 
requirements. In addition, the results suggested an increase 
in rate up to plant capacity can lead to a decrease in unit 
labor requirements. 
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C. SURVEY OF RESEARCH DURING THE EIGHTIES 
l. Crouch 

The purpose of this study was to investigate possible 
sources of bias in the slopes of progress functions which are 
conventionally estimated using the unit cost progress 
function. Crouch [Ref. 14] asserted that given the fact that 
unit costs are a function not only of cumulative output but 
also of the rate of output per time period, the conventional 
unit-cost progress function has omitted variables. Under 
these circumstances, the use of ordinary least squares 
regression may introduce specification bias into the 
estimates. Crouch confirmed this situation mathematically. 

When a variable from the true relation is omitted, a part 
of its influence in explaining the movements of the 
dependent variable is captured by the independent 
variables which are included. When the omitted variable 
is not correlated with any of the independent variables, 
the coefficients of the included variables are not biased. 
[Ref. 14:p. 42] 

Crouch conducted a pilot study to investigate the 
existence of bias when progress functions are estimated in the 
conventional manner. Unit-cost data (on an annual average 
basis) in constant dollars for ten components of the Hawk 
missile were used. The results of this study indicated that 
when progress functions are estimated in the conventional 
manner, biased estimates of the slopes may be obtained in a 
significant number of cases. In particular, the results 
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indicated that when returns to scale are constant, the 
estimated learning curve exponent obtained from the 
conventional progress function will be unbiased. However, 
when the returns to scale are not constant, the estimated 
learning curve exponent obtained from the conventional 
progress function will be biased. Negative bias will occur 
when returns to scale are increasing; positive bias will occur 
when returns to scale are decreasing. 

2 . Smith 

The objective of this study was to examine the impact 
of production rate changes on the unit cost of weapons 
systems. Smith [Ref. 15] provided a summary of the 
significant research on the relationship between production 
rate and weapon system cost. In doing so, he analyzed the 
various findings and conclusions and assessed their 
applicability. The results of his research review indicated 
that only rather weak conclusions could be drawn from the 
existing state of knowledge. The principal findings were: 
production rate affects unit costs but, in most cases, not as 
strongly as the learning (cumulative quantity) effect, and; 
the rate effect varies with the weapon system. Smith 
identified four principal cost-rate models — Womer [Ref. 16], 
Washburn [Ref. 17], Linder and Wilbourn [Ref. 10], and Fazio 
and Russell [Ref. 18]. Each of these models differed in the 
concept of rate, the number of parameters to estimate, and the 
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range of applicable programs. Nevertheless, none of them was 
considered suitable for use by top level budget planners. 
[Ref. 15:p. ii] 

Case studies were conducted using production data for 
six missile systems. These systems possessed a wide range of 
production characteristics ranging from low volume, labor 
intensive to high volume, highly automated production. The 
results of the empirical research supported the belief that 
under program stretch-out, the most important contributor to 
increased unit costs is an increased overhead allocation. In 
addition, the idea that labor inefficiency is often a 
relatively unimportant factor in rate adjustments was 
supported. [Ref. 15:pp. 43-44] Based on the results of the 
case studies, Smith concluded that: 

1. A simple rate-sensitive model which focuses on the effect 
of rate changes on overhead is appropriate for the 
programming and budgeting phases. 

2. Long-range planners should disregard rate behavior and 
focus only on military requirements. In long-term 
planning, production rates and their effects are both 
unpredictable and much less important than other more 
fundamental considerations. [Ref. 15 :p. 46] 

3. Balut 

Standard use of learning curve theory involves an 
implicit assumption that overhead is 100 percent variable with 
direct costs. However, plant overhead is actually comprised 
of three components — variable overhead, fixed overhead, and 
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semi-fixed overhead. Variable overhead costs vary with the 
activity rate. They include production-related indirect costs 
that are tied to the number of direct laborers working in the 
plant and the number of units being produced. Fixed overhead 
costs do not vary with the activity rate and are fixed in the 
short-term (e.g., depreciation, insurance, rent, security). 
Semi-fixed overhead is indirect expenses that are partially 
fixed and partially variable such as utilities. Semi-fixed 
overhead costs are typically gathered and reported as pools; 
consequently, the fixed and variable portions are not 
discernable. [Ref. 19:pp. 63-65] 

The costing of alternative aircraft procurement 
quantities and rates within the Office of the Secretary of 
Defense (Program Analysis and Evaluation) is a two step 
process which considers the heterogeneous nature of overhead. 
First, new average unit prices are derived for each lot, 
consistent with new lot quantities, using the learning curve. 
Then, in order to correct for the erroneous underlying 
assumption in step one, prices are adjusted to reflect the 
redistribution of fixed overhead resulting from a change in 
the production rate. This second step is referred to as rate 
adjustment. [Ref. 19:p. 65] 

The objective of Balut's [Ref. 19] study was to 
evaluate the rate adjustment model used by the Office of the 
Secretary of Defense by comparing its predictions to actual 
contractor performance. An improved version of the model 
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derived using actual contractor data was presented. Balut 
illustrated the use of this improved model for situations when 
the contractor has only one program, and for situations when 
the contractor has other ongoing programs. 

4 . Bemis 

Unit costs for weapons systems have traditionally been 
projected using the experience curve. This method expresses 
the projected unit cost as a function of cumulative quantity 
produced, regardless of the production rate. Prior research 
into the production rate-cost relationship for weapons systems 
indicates that unit cost varies significantly as a function of 
production rate. These variations are, to a great extent, due 
to the amortization of fixed overhead. [Ref. 20:pp. 84-85] 

Bemis [Ref. 20] proposed a method for estimating 
rate/cost/quantity relationships using system specific cost 
estimates. Only unit-fly-away costs were considered. The 
input data for the model was historical rate/cost/quantity 
data for ongoing programs, and contractor or in-house 
estimates for new programs. An equation was derived by 
regressing unit cost on cumulative quantity and production 
rate. In most of the cases analyzed in this study, a high 
multiple correlation coefficient (greater than 0.9) was 
obtained. 

Bemis found that when the production rate was stable, 
the experience curve method and the rate/cost/quantity method 
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generated identical unit cost estimates. However, when the 
production rate was variable, lower unit costs were associated 
with higher production rates, and higher unit costs were 
associated with lower production rates. 

Bemis suggested the rate/cost /quantity model could be 
an invaluable tool for approaching "what if" questions in the 
planning and budgeting process. Moreover, he suggested this 
model offered users a means for readily assessing the cost 
effects of program stretchouts, the costs of maintaining a 
warm production base, and the probable effects of program 
acceleration. 

D. SURVEY OF RESEARCH DURING THE NINETIES 
1. Boger and Liao 

In an effort to reflect the effect of production rate 
on the cost of weapons systems, researchers have proposed a 
variety of adjustments to weapons systems cost models. The 
most popular approach has been to augment the traditional 
learning curve by adding a rate term. The resulting learning 
curve is referred to as a rate adjustment model. [Ref. 21:p. 
32] Boger and Liao [Ref. 21] examined the effects of 
different rate measures and cost structures on rate adjustment 
models and illustrated how alternative surrogate production 
rate measures might lead to erroneous conclusions. 

The effect of production rate on unit cost stems from 
economies of scale. As production rates are increased, 
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facilities are utilized more fully and greater specialization 
of labor occurs. Materials costs are reduced because the 
increased volume of materials purchased results in quantity 
discounts. Finally, the increased production volume allows 
fixed overhead charges to be spread over a larger quantity of 
output. Together, these underlying effects work to increase 
efficiency and lower production costs. [Ref. 21:p. 83] 

Increases in production rate are normally expected to 
result in lower unit costs due to economies of scale; however, 
production rate increases may also lead to diseconomies of 
scale. Such is the case when a plant is operating beyond its 
efficient capacity level. Under these circumstances, factors 
such as over-time pay, lack of skilled labor or the need to 
invest in additional tooling and/or facilities may lead to 
inefficiencies and increased unit costs. [Ref. 21:pp. 23-24] 
Because of the difficulty associated with measuring 
production rate, a number of alternative surrogate measures 
have been adopted. The two primary surrogate production rate 
measures are lot size and annual/monthly production quantity. 
Unfortunately, there are weaknesses associated with the use of 
each of these measures. These weaknesses are as follows: 

1. Lot Size : The time required to produce successive, 

comparably-sized lots frequently changes over the life of 
a program. As a result, it is unclear what is being 
measured by the lot size proxy. 

2. Production Ouantitv/Time Interval : If there is a large 

amount of work-in-progress and the production period is 
long compared to the observation period, units produced 
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in the following time period will actually reflect work 
performed in the previous time period. This can result 
in substantial bias in estimation. 

3. Average Rate for Each Program : This approach may 

understate the effect of descriptive rate changes. An 
average rate for each program is usually used in cross- 
sectional analysis because the production rate may change 
in a typical production run. 

4. Cumulative Quantity : Cumulative quantity is highly 

correlated with each of the surrogate production rate 
measures discussed above. As a result, analysts have 
been unable to separate statistically the effect of 
learning and production rate. [Ref. 21:pp. 86-87] 



In order to avoid some of the difficulties associated 
with using these surrogate production rate measures, Boger and 
Liao recommended a ratio of these measures be used. This 
ratio should be keyed to a base production rate. Adoption of 
this approach offers a number of advantages: 



1. Using a surrogate production rate ratio tends to mitigate 
the multicollinearity problem. 

2. Using the rate to which the manufacturer has tooled as 
the base rate provides an indicator of returns to 
variable inputs. Ratios greater than one indicate 
decreasing returns while ratios less than one indicate 
increasing returns. [Ref. 21 :p. 88] 



In addition to examining the problems associated with 
the two primary surrogate rate measures, Boger and Liao 
examined the problem of changing cost structures. These 
changes occur as a result of changes in the production setup. 
Based on their analysis, the researchers concluded that rate 
adjustment models are appropriate only when applied to data 
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collected from plants which have not undergone changes in cost 
structure. 

2. Moses 

In 1991, Moses [Ref. 5] published the results of a 
study in which he investigated and compared the forecasting 
bias for the learning curve model and a rate adjustment model. 
Specific objectives of the study were as follows: to 
determine if either the learning curve model or rate 
adjustment model exhibits consistent/systematic bias; to 
determine under what circumstances the two models are biased, 
and; to identify the nature of the bias (i.e., overestimation 
or underestimation of future costs) . 

A simulation approach was used to conduct the 
research. First, cost series were generated under varying 
simulated conditions. Then, model parameters were estimated 
by fitting the learning curve and rate adjustment models to 
the cost series. Future costs were predicted using each of 
the models. These predicted future costs were then compared 
with the actual cost to measure bias. Finally, the 
relationship between the level of bias and the simulated 
conditions was investigated using analysis of variance. 

The simulation was conducted by varying seven factors 
which had been found to affect the magnitude of model 
prediction errors in prior research. These seven factors 
were : 
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1. Data Hi story - -the number of data points available to 
estimate the parameters for a model. 

2. ’ariable Cost Learning Rate --the learning curve exponent. 

3 . Fixed Cost Burden — the proportion of total cost comprised 
of fixed costs. 

4 . Production Rate Trend — the production trend during the 
model estimation period (i.e., gradual growing trend or 
level trend) . 

5. Production Rate Instabilitv/Variance — period-to-period 
fluctuations in the production rate perhaps caused by 
changes in the demand for output or the supply of inputs, 
and annual budget uncertainties. 

6. Cost Noise Variance — variability in period-to-period 
cost — designed to reflect unsystematic, unanticipated, 
non-recurring random factors (e.g., changes in the cost, 
type or availability of input resources, temporary 
variations in the level of efficiency, and unplanned 
changes in the production process) . 

7 . Future Production Level — the production rate planned for 
the future relative to past levels. 



Bias was measured separately for each model as follows: 



BIAS = 



PUC - AUC 

AUC 



where 

PUC = Predicted unit cost either the learning curve or 
the rate adjustment model. 

AUC = Actual unit cost generated by the cost function. 

In conducting his analysis, Moses found that the rate 
adjustment model provided unbiased cost estimates while the 
learning curve consistently underestimated actual costs. The 
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following conclusions regarding learning curve bias were 
drawn : 



1. Learning curve bias stems from the fact that a portion of 
total cost is fixed. The log linear relationship between 
cost and quantity assumed by the learning curve does not 
hold when fixed costs, which are not subject to learning, 
are present. 

2. Bias increases as the proportion of total cost made up of 
fixed costs increases. This relationship holds up to the 
point where fixed costs account for 50 percent of total 
cost; further increases above that level reduce bias. 

3 . The production rate during the period of model estimation 
and the production rate during the period for which costs 
are forecast both affect the degree of bias. Bias is 
minimized when there is a consistent production trend 
during these periods. Bias is magnified when there is a 
shift in production rate trend. 

4. The steeper the learning curve slope, the greater the 
level of bias. (This conclusion is based on the 
assumption that the proportions of total cost that are 
fixed or variable remain relatively stable.) 

5. The greater the number of observations, the higher degree 
of bias. 

6. The further into the future predictions are made, the 
greater the degree of bias. 



E . SUMMARY 

The review of previous research conducted in this chapter, 
while comprehensive, is by no means all-inclusive. 
Nevertheless, the studies discussed do provide a sound basis 
for assessing the current level of understanding with respect 
to learning curves and rate adjustment models. The following 
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conclusions can be drawn concerning the research efforts to 
date : 



1. The broad objective of most of the studies dealing with 
extensions or modifications to the learning curve has 
been to investigate the relationship between production 
rate and cost to determine whether or not consideration 
of production rate leads to improved cost estimates. 

2. Findings concerning the importance of considering the 
effect of production rate on unit costs when predicting 
weapons systems costs have been inconclusive. Some 
studies have found production rate to be an important 
predictor of cost variation while other studies found 
production rate to be of little or no significance. 

3. Production rate affects unit cost but, in most cases, not 
as strongly as the learning (cumulative quantity) effect. 

4. The production rate effect varies with the weapon system 
and the specific cost elements. 

5. The influence of production rate on unit cost depends on 
the relative amounts of direct and indirect costs per 
unit and the proportion of each category that can be 
considered as fixed or variable. 

6. Variations in unit cost in response to production rate 
changes stem from economies of scale. These variations 
are, to a great extent, due to the amortization of fixed 
overhead. 

7. Lower unit costs are associated with high production 
rates and higher unit costs are associated with lower 
production rates. 

3. The rate adjustment model, as described by Moses (1991), 
provides unbiased cost estimates. Conversely, the 
learning curve consistently underestimates actual costs. 

9. Learning curve bias stems from the fact that a portion of 
total cost is fixed. 

10. Learning curve bias is affected by the proportion of 

total cost made up of fixed costs, the production rate 
during both the period of model estimation and the period 
for which costs are forecast, the slope of the learning 
curve, the number of observations, and the time horizon 
for which predictions are being made. 
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Findings by Moses [Ref. 5] concerning the bias associated 
with the traditional learning curve and the rate adjustment 
model are noteworthy. Nevertheless, their significance is 
somewhat tempered by the use of simulated data. The remainder 
of this thesis will focus on confirming or disconf irming the 
results of the Moses [Ref. 5] study and will extend the 
research to include two additional models — a random walk model 
and a model incorporating production rate effects. 
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III. SAMPLE, DATA AND VARIABLES 

The purpose of rhis chapter is to provide an overview of 
the methodology used to conduct this study. The chapter will 
begin with a description of the population from which the 
sample was drawn, the criteria for inclusion of a weapon 
system/procurement program in the sample and the specific raw 
data collected for each weapon system. This will be followed 
by a description of the procedure used to "expand" the sample. 
Next, the four cost progress models that were included in the 
study will be introduced along with the procedure used for 
predicting costs with these models. Finally, the procedure 
used to measure bias in the study will be explained and the 
"demographic variables" and the "condition variables" that 
were selected as likely candidates for being significant 
explainers of bias will be defined. 

A. SELECTION OF THE SAMPLE 
1. Data Sources 

Data used in conducting the study were obtained from 
two sources: the U.S. Military Aircraft Cost Handbook (Ref. 
22], and; the U.S. Missile Cost Handbook [Ref. 23]. Data 
contained in these handbooks were based on historical Service 
cost data and reflected the annual total obligational 
authority (TOA) flyaway costs for the included programs. 
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Aircraft data fell into five major categories: attack 
aircraft, fighter aircraft, bombers, attack helicopters, and 
patrol aircraft. Missile data fell into four main categories: 
air-to-air missiles, air-to-surf ace missiles, surface-to-air 
missiles, and surface-to-surface missiles. All cost data 
contained in these handbooks was normalized to a constant 
fiscal year base (FY-81) . Consequently, the consistency of 
this data was ensured for all aircraft and missile programs. 

Weapons systems contained in the aircraft handbook 
were U.S. Navy, Air Force, Marine Corps and Army aircraft. 
These aircraft were combat-oriented and were in the active 
U.S. inventory during the FY 1960-1980 period. Trainers, 
reconnaissance or electronic warfare variants and those 
aircraft produced for foreign military sales were not included 
in the handbook. 

The missile handbook contained cost information for 
U.S. Air Force, Navy, and Army missile programs during the 
period FY 1961-1983. 

Costs reflected in these handbooks were TOA dollars — 
the amounts budgeted in a specific fiscal year. TOA dollars 
do not reflect actual expenditures in any given fiscal year. 
Nevertheless, TOA dollars do provide an excellent proxy for 
actual dollar expenditures because it is customary within the 
DoD to ensure that expenditures match total obligational 
authority prior to the lapsing of an appropriation. As a 
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result, differences between TOA and actual expenditures are 
normally very small. 

2. Selection Process and Criteria 

Selection of aircraft and missile programs for 
inclusion in this study was based on three criteria: the 

number of plot points or fiscal years, the availability of 
airframe cost data and the completeness of the data. Only 
programs for which five or more fiscal years worth of data was 
available were included in the study. This criterion was 
required to ensure the minimum amount of data necessary for 
statistically fitting the cost progress models. The latter 
two criteria were established to ensure that meaningful 
analyses could be conducted. As a result, only aircraft 
programs which had complete quantity and airframe cost data 
were included in the study. In addition, only missile 
programs for which complete quantity, and guidance and 
control/airframe costs were available were included in the 
study . 

3. Programs Selected for Study 

Based on the three criteria described above, forty-six 
weapons procurement programs (fourteen missile programs and 
thirty-two aircraft programs) were selected for inclusion in 
this study. The specific aircraft and missile programs that 
comprised the sample are included in Tables 1 and 2 
respectively. 
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TABLE 1 





AIRCRAFT 


PROCUREMENT 


PROGRAMS 


INCLUDED IN 


THE STUDY 


1. 


A-3A/B 


12 . 


AH- IS 


23 . 


F-16A 


2 . 


A-4C 


13 . 


AV-8A 


24 . 


F-101A/B/C 


3 . 


A-4E 


14 . 


B-47B/E 


25. 


F-102A 


4 . 


A-4A/B 


15. 


F-3A/B/C 


26. 


F-105D 


5. 


A-4E/F 


16. 


F-4A 


27. 


F-105B/D 


6. 


A- 6 A 


17 . 


F-4B 


28 . 


F-111F 


7. 


A-6E 


18 . 


F-4E 


29. 


P-2H 


8. 


A-7D 


19 . 


F-4J 


30. 


P-3A 


9. 


A-7E 


20. 


F-4A/B 


31. 


P-3C 


10. 


A-37B 


21. 


F-8D/E 


32 . 


S-3A 


11. 


AH-1G 


22 . 


F-14A 
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TABLE 2 



MISSILE PROCUREMENT PROGRAMS INCLUDED IN THE STUDY 



1 . 


PHOENIX ( AIM-54A) 


8. 


STANDARD ER (RIM-67B/ 
B-l/C-1) 


2 . 


SIDEWINDER (AIM-9D/G) 


9. 


STANDARD ER (RIM-67A) 


3 . 


SIDEWINDER (AIM-9H) 


10. 


STANDARD MR ( RIM-6 6A) 


4 . 


SIDEWINDER (AIM-9H/L) 


11. 


STANDARD MR (RIM-66B) 


5. 


SPARROW ( AIM-7 E) 


12 . 


TALOS ( RIM-8 E) 


6 . 


SPARROW (AIM-7F) 


13 . 


TARTAR (RIM-2 4B) 


7 . 


SHRIKE (AGM-45A) 


14 . 


TERRIER (RIM-2E) 



TABLE 2 



B. COMPILATION OF PROGRAM DATA 

Once the sample of aircraft and missile procurement 
programs had been selected, the following data were obtained 
for each program in the sample: 



1. Program Name — name of the weapon system. 

2. Manufacturer — name of the prime contractor. 

3. Military Service — identified which Service branch (es) 
procured the weapon system. 

4. Program Type — identified the nature of each weapon 
system, i.e., aircraft or missile. 

5. Mission — identified each weapon system according to its 

primary mission. Aircraft program types were: 

f ighter /attack , fighter, attack, bomber, patrol, and attack 
helicopter. Missile program types were: air-to-air, air-to- 
surface, surface-to-air, and surface-to-surface. 

6. Modification — identified whether a particular weapon 
system was an entirely new design or a modification of an 
existing design. 

7. Combined Program — identified whether a procurement 
program included several versions of a given weapon system 
or only one version. 

8. Fiscal Year — identified the fiscal years during which 
quantities of a particular weapon system were procured. 

9. Quantity — identified the number of units of each weapon 
system procured in a given fiscal year. 

10. Aircraft Airframe Cost/Missile Guidance and Control 
Airframe Cost 
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C. EXPANSION OF THE SAMPLE 

Following the compilation of data described in the 
previous section, the original data sample of forty-six 
programs was partitioned into 121 cost series. This 
"expansion of the sample" was accomplished by dividing each 
program cost series into individual, consecutive year-to-date 
cost series. For example, if a particular procurement program 
had cost data available for seven fiscal years, e.g., FY 1970- 
1976, this single cost series could be expanded into four 
separate cost series as shown below. 



Cost Series # 1: 
Cost Series # 2 : 
Cost Series # 3 : 
Cost Series # 4: 



FY 1970-1973 (used to predict 1974 cost) 
FY 1970-1974 (used to predict 1975 cost) 
FY 1970-1975 (used to predict 1976 cost) 
FY 1970-1976 (used to predict 1977 cost) 



The initial cost series for each weapon system included in the 
sample was comprised of data from the first four fiscal years 
of that particular program (the minimum number of years needed 
to estimate the cost models) . Each subsequent cost series for 
a given program was then created by additionally including the 
data point for the next fiscal year in the existing cost 
series. By partitioning the sample size in the study in this 
manner, it was possible to simulate actual usage of the 
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various cost progress models over time and evaluate their 
performance under varying data availability conditions. 

D. SELECTION OF COST PROGRESS MODELS 

Previous research has demonstrated that three factors are 
particularly useful in predicting the future costs of weapons 
systems: past cost, cumulative quantity, and production rate. 
Numerous models have been introduced in an effort to improve 
the quality of cost estimates over those obtained using the 
traditional learning curve model. These models are 
differentiated by the explanatory variables included and the 
underlying assumptions with regard to the relative importance 
of past cost, cumulative quantity and production rate in 
predicting future costs. 

In addition, these models may be differentiated by the 
period or length of time over which data is observed and used 
in creating a forecast. Some models assume that future cost 
depends only on the most recent cost, quantity and/or 
production rate levels. However, other models assume that 
cost, quantity or production rate levels from early in a 
program's life are also significant and, as a result, 
specifically consider data covering the full production life. 
Regardless of which approach is used, there are both 
advantages and disadvantages associated with the use of data 
comprised of only recent observations or data comprised of 
both recent and earlier observations. 
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Use of recent observations may increase the relevance of model 
results in light of the current situation but sacrifices any 
information reflected in earlier, historical data. In 
addition, models based only on a few recent observations are 
more susceptible to random noise. Conversely, use of older 
observations may reduce the relevance of results or lead to 
results that are not representative of the current situation. 
However, the use of additional data reduces the impact of 
random variance in recent observations. 

Finally, in addition to the two factors discussed above, 
cost progress models may be differentiated by the form of the 
assumed relationship between the dependant variable — cost--and 
the potential explanatory variables — past cost, cumulative 
quantity, and production rate. Relationships may be linear, 
log linear or some other form. 

Four alternative models were selected for inclusion in 
this study: a random walk model, the traditional learning 
curve model, a model which expresses unit cost as a function 
of past cost and production rate (the fixed-variable model) 
and the common rate adjustment model. 

1. Random Walk Model 

The random walk model assumes that future cost is a 
function of past cost; however, only the most recent cost is 
relevant. Any deviation from predicted cost is considered 
random deviation. 
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The model is expressed as follows: 



UC t = UC t 



where 

UC t = Predicted unit cost in time period t. 

UC t . 1 = Actual unit cost in time period t - 1 . 

t = Time period. 

This model was selected for inclusion in the study for two 
reasons. First, it exemplifies the cost estimation method 
used by budget programmers when there is only very limited 
historical data. Under these circumstances, future cost 
projections are often based on actual costs in the previous 
period. Second, the random walk model is the most basic and 
naive cost estimation model and, as such, provides a useful 
benchmark for evaluating the performance of other more 
sophisticated models. 

2. Traditional Learning Curve Model 

The second model selected for inclusion in the study 
was the traditional learning curve. This model assumes that 
future cost is a function of both past cost and cumulative 
quantity produced. Moreover, all historical cost data is 
considered to be relevant. 
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The traditional learning curve model assumes a log linear 
relationship between cost and cumulative quantity and is 
expressed as follows: 

UC, = aQc 



where 

UC t = Incremental unit cost of item at quantity Q. 

Q c = Cumulative quantity produced as of period t. 

a = Theoretical first unit cost. 

. b = = Learning curve exponent . 

r - Learning rate. 

The traditional learning curve model was selected for 
inclusion in the study because it is the most widely 
researched cost progress model and represents the foundation 
on which other cost progress model variants are based. 

3. Fixed-Variable Model 

The third model selected for inclusion in the study 
was the fixed-variable model. This model specifically 

addresses the relationship between total unit cost and unit 
variable costs and unit fixed cost. Variable cost per unit 
remains constant; however, fixed cost per unit varies 
depending on the production volume because total fixed cost is 
allocated by spreading it over the total volume of output. 



42 



The fixed-variable model is expressed as follows: 



UC t = a 




where 

UC t = Unit cost in period t. 
a = Variable cost per unit, 
b = Standard fixed cost per unit. 

PR t = Production rate = Q t * Q AVG . 

Q t = Production quantity in period t. 

Q avg = Average production quantity per period. 

This model was included in the study because it explicitly 
considers the impact of production rate on unit cost through 
the allocation of fixed overhead. In contrast to the 
traditional learning curve model (which includes cumulative 
quantity but not production rate as an explanatory variable) , 
the fixed-variable model includes production rate but ignores 
cumulative quantity as an explanatory variable. 

4. Bemis Production Rate Adjustment Model 

The final model included in the study is the most 
widely used rate adjustment model. This model, popularized by 
Bemis [Ref. 16], was developed by augmenting the traditional 
learning curve model with a production rate term. 
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The model is expressed as follows: 



UC, = aQf Rt 



where 



UC t = Predicted unit cost at quantity Q and production rate 
per period R. 

Q t = Cumulative quantity produced as of period t. 

R t = Production rate in period t. 
a = Theoretical first unit cost, 
b = Learning curve exponent . 
c = Production rate exponent . 



The Bemis production rate adjustment model was included in the 
study because it considers both cumulative quantity and 
production rate (in addition to past cost). Hence, it is the 
most comprehensive of the four models in the study. 



E. ANALYSIS OF BIAS 

1. Unit Cost Prediction 

In order to assess the bias exhibited by the random 
walk model, the traditional learning curve model, the fixed- 
variable model, and the Bemis model, predicted unit costs were 
estimated by applying each of the four models, in turn, to the 
actual cost series. The following paragraphs describe how 
this procedure was accomplished. 

The random walk model assumes that unit cost in the 
next period is the same as unit cost in the current period. 
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Consequently, the predicted unit cost for any period (t) was 
simply the actual cost from the preceding period (t-1) . 

Predicted unit costs for the remaining three models 
were derived in the following manner. First, each of the 
models was separately fit to the initial cost series 
(comprised of the first four fiscal years' data) for each 
weapon system to derive the models' parameters. Then, the 
appropriate data values for cumulative quantity and/or 
production rate from period five were input into each of the 
models to obtain estimated unit costs for period five. These 
two steps were then repeated for each remaining cost series 
for the various weapons systems until predicted costs had been 
derived for every fiscal year for which actual cost data was 
available (e.g., models were next estimated on five years 
worth of data, then used to predict the cost for year six). 
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2 . Bias Measurement-The Dependant Variable 

Once predicted unit costs had been computed for each 
cost series with each model, a measure of bias was determined 
for each prediction as follows: 



BIAS = 



PUC - AUC 

AUC 



where 



BIAS = Percentage difference between predicted unit cost 
and actual unit cost. 

PUC = Predicted unit cost from the particular model of 

interest . 

AUC = Actual unit cost obtained from original unit cost 
data . 



Positive BIAS values indicate a model has overestimated actual 
future cost; negative BIAS values indicate a model has 
underestimated actual future cost. BIAS values of zero 
indicate the predicted cost and actual future cost are 
identical and the associated model is unbiased. 

As in the earlier study conducted by Moses [Ref. 5], 
BIAS represented the dependent variable in the statistical 
analysis. The basic objective of the study was to determine 
what factors or conditions are useful in explaining variance 
in BIAS. 
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The measures of bias and associated labels for each of the 



four models were as follows: 

BIASRW 
BIASLC 
BIASFV 
BIASBE 



Bias for the Random Walk Model: 
Bias for the Learning Curve Model: 
Bias for the Fixed-Variable Model: 
Bias for the Bemis Model: 



Given 121 cost series (from the 46 procurement programs), 
there were 121 separate measures of bias for each model. 

3. Explanatory Variable Selection 

Model performance in prediction (i.e., the degree and 
direction of bias) depends on the circumstances in which the 
model is used. Two broad categories of factors which might 
influence model performance and, hence, be useful in 
explaining bias, were identified in the study. The first 
category consisted of "demographic variables" and the second 
category consisted of "condition variables". Together, these 
two groups of variables were the independent variables in the 
statistical analysis. Table 3 summarizes the independent 
variables and their corresponding labels. 
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TABLE 3 



INDEPENDENT VARIABLES 



Demographic Variables 
Program Type 
Modification 
Military Service 
Mission 

Condition Variables 
Burden 

Cost Variance 
Learning Rate 
Production Rate Variance 
Future Production Level 
Past Production Trend 
Plot Points 



Labels 

TYPE 

MOD 

MILSERV 

MISSION 

BURDEN 

CVAR 

LRATE 

RATEVAR 

FUTUPROD 

BEGTREND , 

PLOTPNTS 



ENDTREND 
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a. Demographic Variables 

The demographic variables describe characteristics 
of the weapons procurement program. Four demographic 
variables were considered in the study: 

1 . Program Type 

2. Modification 

3. Military Service 

4 . Mission 

These particular demographic variables were 
selected for investigation because they represent readily 
apparent characteristics of the various programs which may 
affect model bias. 

b. Condition Variables 

In addition to the four demographic variables, 
seven condition variables were considered in the study: 



1 . 


Burden 




2 . 


Cost Variance 




3 . 


Learning Rate 




4 . 


Production Rate 


Variance 


5. 


Future Production Level 


6. 


Past Production 


Trend 


7 . 


Plot Points 
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The label "condition variables" is used here because each of 



these variables in some manner indicates something about 
conditions that existed during the weapon system procurement. 
These are analogous independent variables (factors) to those 
examined by Moses [Ref. 5]. The following paragraphs describe 
each of these variables along with the underlying rationale 
for their inclusion in the study. 

(1) Burden 

Burden (BURDEN) indicates the percentage of 
total unit cost made up of fixed cost. Burden is measured as 
follows : 

BURDEN = — £— ■ 
a + j b 



where 

a = Variable cost per unit ( constant ) . 
b = Standard fixed cost per unit. 

Note: a and b were estimated parameters from the fixed- 

variable model. 

Burden was included in the study because past research (Linder 
and Wilbourn [Ref. 10], Moses [Ref. 5] and Smunt [Ref. 24]) 
has shown that burden directly influences the impact of 
changes in production rate on unit cost. As production rate 
increases, the cumulative quantity produced during a period 
increases and the variable cost per unit decreases due to the 
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incidence of learning. In addition, as production rate in a 
period increases, fixed cost per unit, is reduced as total 
fixed cost is spread over a larger production output. The 
impact of production rate increases as the proportion of total 
cost made up of fixed costs increases. As a result, the 
relative bias of the various models may depend on the level of 
burden . 

(2) Cost Variance 

Cost variance (CVAR) indicates the amount of 
unsystematic variation in unit cost that may result from 
unanticipated, non-recurring, random factors. Examples 
include changes in the cost, type or availability of input 
resources, temporary fluctuations in efficiency, and unplanned 
changes in the production process. 

By assessing the amount of unsystematic 
variation associated with various weapons procurement 
programs, it may be possible to determine whether there is a 
relationship between cost stability and bias. For this 
reason, cost variance was examined in the study. 
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Cost variance was measured as follows: 



CVAR 



E l C t - ^v G i 

r 

VaVG 

n 



where 

CVAR = Cost variance . 

C t = Unit cost for a given production period. 

C AVG = Average unit cost for all production periods to date. 

n = Total number of production periods. 

(3) Learning Rate 

Learning rate (LRATE) measures the decrease in 
per unit cost that occurs as the quantity of units 
manufactured doubles. It is affected by the type of 
production process and the complexity of the product design. 
Smunt [Ref. 24] found that the degree of learning in the 
underlying production process determines the improvement in 
prediction accuracy that results from including a learning 
parameter in a model. Learning rate was examined in the study 
to determine the nature of the relationship between learning 
rate and bias for the various models. 
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Learning rare was measured as follows: 



where 



r = Learning race expressed as a percencage. 
b = Learning curve exponent as estimated 

by the traditional learning curve model. 



(4) Production Rate Variance 

Production rate variance (RATEVAR) reflects the 



severity of period-to-period fluctuations in production rate. 
These fluctuations may result from either changes in demand 
for a particular weapon system or changes in the cost or 
availability of production inputs. Production rate variance 
was measured as follows: 



RATEVAR = 



y i^c Qavg\ 

Q AVG 



n 



where 



RATEVAR 



n 




Rate variance . 

Quantity of units produced in the current period. 
Average quantity of units produced for all 
periods to date. 

Total number of periods to date. 
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(5) Future Production Level 

Future production level (FUTUPROD) indicates 
whether the level of production in the next period (the period 
for which cost is to be forecast using the model) is high or 
low relative to the current period. Future unit costs are 
predicted using cost progress models fit to past production 
data. As a result, the accuracy of the various models may be 
affected if the production level in the period for which costs 
are being estimated varies significantly from the production 
levels that existed during previous periods. Future 

production level was examined in the study to determine 
whether production growth and production cutbacks affect the 
tendencies of the various models to over/underestimate unit 
costs . 

Future production level was measured as 

follows : 



Q„ 

FUTUPROD = log (— £) 
~ Cl 



v/here 



FUTUPROD = Future production level. 

Q n = Production level for the next period for which 

costs are being forecast . 

Q l - Production level for the last ( most recent) 

period. 
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{ 6) Past Production Trend 

Past production trend indicates the pattern of 
production volume associated with each weapon system included 
in the study. Two variables, 3EGTREND and ENDTREND , were used 
to reflect how production volume was changing at the beginning 
and end of each cost series to which models were fit. 
Production rate per period may be low initially in order to 
work out bugs and ensure a stable product design prior to full 
scale production. Alternatively, initial production rate per 
period may be high if the current weapon system represents an 
updated version of an already existing system with only minor 
modifications . 

Production rate per period may also vary at the 
end of the production run depending on whether the program is 
abruptly cancelled, gradually phased out or continued at some 
minimum level in order to maintain a warm production base. 
These two production trend variables were included in the 
study to determine whether the bias associated with the 
various models was related to past production trend. 
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The production trend variables were measured as follows: 



and 



SEGTREND = 



five: C'i 



•AVG 



ENDTREND = 



2 , - Q. 



AVG 



'■AVG 



where 



3EGTREND = Production trend at the beginning of 
the production run. 

ENDTREND = Production trend for the most recent production 

period. 

Q x = Quantity produced in first production period. 

Q l = Quantity produced in most recent period. 

Q AVG = Average quantity per period produced through 

current period. 



Positive values for BEGTREND and ENDTREND indicate production 
trends that are increasing in volume. 

(7) Plot Points 

The final independent variable included in the 
study was the number of plot points (PLOTPNTS) . Plot points 
indicates the number of data points available to estimate 
model parameters. The accuracy of the learning curve, 
fixed-variable, and Bemis models should improve as the amount 
of data available during the model estimation period 
increases. Nevertheless, if a model is inherently biased, 
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increasing the number of plot. points will nor necessarily 
eliminate the bias associated with the model. The number of 
plot points was considered in the study to determine whether 
or not the bias associated with the various models could be 
explained in terms of the availability of past production 
data . 

F. SUMMARY 

This chapter has provided an overview of the sample and 
variables to be used to investigate bias. The discussion 
began with a description of the sample selection process, and 
a summary of the data collected for each program included in 
the sample. This was followed by a description of the 
procedure used to expand the sample from 46 observations 
(programs) to 121 cost series. Next, the four cost progress 
models included in the study — the random walk model, the 
traditional learning curve model, the fixed-variable model, 
and the Bemis production rate adjustment model were described 
along with the rationale for their selection. The cost 
estimation procedure was then explained and a method for 
measuring bias was introduced. Finally, two categories of 
independent variables — demographic variables and condition 
variables — were introduced to be evaluated as potential 
sources of model bias. 
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IV. ANALYSIS AND FINDINGS 



Chapter III introduced eleven variables — four demographic 
variables and seven condition variables — which might influence 
the performance of alternative cost progress models and, 
hence, be useful in explaining the cost prediction bias of 
these models. This chapter describes the statistical 
procedures that were performed to assess the significance of 
each of these variables in explaining bias. It will begin by 
providing an overview of the statistical tests that were used 
to analyze the variables. This will be followed by a 
presentation of some general findings with respect to the 
performance of the four alternative cost progress models. 
Next, model specific findings regarding the significance of 
each of the eleven explanatory variables will be presented. 
Finally, the results of the analysis will be summarized and 
conclusions will be presented. 

A. STATISTICAL PROCEDURES 

Prior to conducting statistical analyses, the distribution 
of values for each of the dependent and independent variables 
was assessed. The presence of extreme values within the data 
set could unduly influence the outcomes of the statistical 
analyses. Accordingly, variable values which lay beyond three 
standard deviations from the mean for the variable were 
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truncated. Truncation involved replacing extreme values with 
values equal to three times the standard deviation for the 
appropriate variable. This approach was applied because it 
reduces the influence of outliers on the results of the 
statistical analyses without discarding and, consequently, 
ignoring the impact of these observations. 

Once all extreme values had been identified and truncated, 
the dependency of BIAS on the demographic variables was 
evaluated for each model using analysis of variance (ANOVA) . 
After determining the significance of the demographic 
variables in explaining BIAS, the significance of the seven 
condition variables and MOD was assessed using both simple and 
multiple linear regression analysis. Finally, three different 
sets of Spearman and Pearson correlation analyses were 
conducted. First, correlation coefficients were computed for 
the measures of BIAS from the four different models. Next, 
correlations between the condition variables (including MOD) 
and BIAS were examined for each model. Then values for the 
condition variables (including MOD) were correlated with each 
other in an effort to detect potential multicollinearity . 

In conducting tests of statistical significance, findings 
with alpha values less than 0.01 were considered significant. 
When analyzing pairwise correlations between the explanatory 
variables, correlation coefficients larger than 0.50 were 
regarded as offering strong evidence that multicollinearity 
might be a problem. 
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B. ANOVA RESULTS 



Analysis of variance was performed for each of the four 
cost progress models to determine whether or not prediction 
bias could be explained by any of the demographic variables. 
The following demographic variables were examined: 

1. TYPE 

2. MILSERV 

3. MISSION 

4 . MOD 

5. TYPE x MOD (interaction variable) 

None of the demographic variables tested were significant in 
explaining the bias of cost predictions made with the random 
walk, fixed-variable or Bemis models. However, MISSION, MOD 
and the TYPE x MOD interaction variable were all found to be 
significant in their ability to explain variations in the bias 
of learning curve cost predictions. ANOVA results for the 
learning curve model are provided in Table 4. Findings 
concerning the significance of MOD in explaining variations in 
the bias of learning curve cost predictions will be discussed 
later, along with the regression results. Differences in 
learning curve bias due to MISSION and TYPE are simply noted. 
No hypothesis was offered to expect differences in learning 
curve cost prediction bias in relation to MISSION and TYPE. 
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LEARNING CURVE MODEL COST PREDICTION BIAS 
ANALYSIS OF VARIANCE RESULTS 
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Consequently, findings related to these factors were not 
pursued further but are mentioned here for possible 
investigation in future research. Based on the fact that the 
utility of these variables in explaining model bias was 
limited to the learning curve model, the remainder of the 
study was devoted to examining the significance of the 
condition variables (including MOD) in explaining model bias. 

C. OVERALL MODEL PERFORMANCE 
1. Mean Bias 

Summary results for the average level of bias 
associated with cost predictions made using the random walk, 
learning curve, fixed-variable and Bemis models are provided 
in Table 5. The mean bias of predicted unit costs estimated 
with the four models ranged from -0.008313 for the learning 
curve to 0.375045 for the fixed-variable model. The results 
show that on average, the learning curve underestimated unit 
costs by approximately 0.83% while the Bemis, random walk and 
fixed-variable models overestimated unit costs by 
approximately 2.9%, 4.6% and 37.5% respectively. Bias 
measures for all four models were skewed in the positive 
direction. Hence, measures of the median bias were examined 
in an effort to obtain a more objective assessment of model 
performance . 
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TABLE 5 



RANK ORDERING OF MODELS BY MAGNITUDE 
OF POSITIVE BIAS (Lowest to Highest) 



If 

Model 


Mean Bias 


Median 

Bias 


Standard 

Deviation 


Skewness 


Learning 

Curve 


-0 . 008313 


-0.060802 


0.335325 


3 . 708457 


Bemis 


0.029472 


-0.012925 


0 . 339456 


0.838916 


Random 

Walk 


0 . 045842 


0.015789 


0 . 199953 


1 .396772 


Fixed- 

Variable 


0.375045 


0.188371 


0 . 929000 


3 . 391860 



63 



Median Bias 



2 . 

The median bias of predicted unit costs estimated with 
the four models ranged from -0.060802 for the learning curve 
to 0.188371 for the fixed-variable model. When the median 
bias measurements for the various models were compared, the 
relative ordering of the models according to magnitude of 
positive bias remained the same. However, the magnitude of 
positive bias decreased significantly in all cases. This 
suggests that some observations had a very large positive 
bias, causing the mean bias to be more positive than the 
median. The direction of bias remained unchanged for all 
models except for the Bemis model. In this case, mean model 
bias was approximately 2.9% while median model bias was 
approximately -1.3%. These results seemed to indicate that 
the learning curve and Bemis models underestimate unit cost 
(provide low unit cost estimates) while the random walk model 
and especially the fixed-variable model overestimate unit cost 
(provide high unit cost estimates) . Perhaps not 
coincidentally, both the learning curve model and the Bemis 
model use cumulative quantity to predict future cost, while 
neither the random walk model nor the fixed-variable model 
contain a cumulative quantity term. 
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3 . Model Bias Correlations 

Both Pearson correlation coefficients and Spearman 
correlation coefficients were computed for the values of bias 
associated with each of the four cost progress models. The 
results of these correlation analyses are provided in Table 6. 
All Spearman correlation coefficients were positive and 
significant; correlation values ranged from 0.27220 to 
0.58048. However, these results were not completely confirmed 
by the computed Pearson correlation coefficients. Values 
obtained from the Pearson correlation analysis indicated 
significant bias correlations for only three of the six paired 
model combinations: BIASRW-BIASLC, BIASRW-BIASBE , and BIASBE- 
BIASFV. Significant Pearson correlation coefficients ranged 
from 0.26401 to .063335. Both correlation analyses indicated 
that the strongest positive correlation existed between random 
walk model bias and learning curve model bias. The second 
highest correlation in both analyses existed between random 
walk model bias and Bemis model bias. The third relationship 
which was correlated and significant in both analyses was 
Bemis model bias and fixed-variable model bias. 

Based on the outcomes from the correlation analyses, 
it was concluded that there is evidence of a positive 
correlation between bias for all paired model combinations. 
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Table 6 



CORRELATION ANALYSES OF MODEL PREDICTION BIAS 



Spearman Correlation Coefficients 





BIASRW 


BIASLC 


BIASFV 


BIASBE 


BIASRW 


1.00000 


0.58048* 


0.27220* 


0.34698* 




O 

o 


0.0001 


0.0025 


0 . 0001 


BIASLC 


0 . 63335 * 


1.00000 


0 . 35518* 


0.43751* 




0.0001 


0. 0 


0 . 0001 


0 . 0001 


BIASFV 


- 0.02073 


- 0.04602 


1.00000 


0.54488* 




0.8215 


0.6162 


0 . 0 


0.0001 


BIASBE 


0 . 26401 * 


0.13296 


0 . 35905 * 


1 . 00000 




0.0034 


0.1460 


0.0001 


0.0 



Pearson Correlation Coefficients 



Correlation Coefficients/Prob>\R\ under H a \ Rho = 0 
* Indicates a <, 0.01 
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That being the case, it seemed reasonable to suspect that, to 
the extent prediction bias was positively correlated among the 
various models, the random walk, learning curve, fixed- 
variable and Bemis models might perform in a similar manner 
under the same circumstances. Moreover, it was possible that 
the significance of various explanatory variables in 
explaining prediction bias might be similar between models 
whose prediction bias was highly correlated. In order to 
resolve these issues, correlation analyses and regression 
analyses were used to study the relationship between model 
bias and the explanatory variables. The following paragraphs 
provide the results of these analyses for each of the four 
models . 

D. REGRESSION ANALYSES AND RESULTS 

The significance of the seven condition variables and one 
demographic variable — MOD — was assessed using regression 
analysis. First, simple regression analyses were performed to 
independently test the significance of these independent 
variables in explaining prediction bias when considered in 
isolation. Then multiple regression analyses were conducted 
to determine the significance of these same independent 
variables in explaining prediction bias while controlling for 
the effects of the other independent variables. Pairwise 
correlations were additionally determined among both 
independent and dependent variables. The following sections 
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describe the results of these analyses for each of the four 
models. The discussion of the results will be organized by 
model type in the following sections. Summary tables 
containing average bias for particular subsets of the sample 
(Table 7) and correlations (Table 8) are provided here. They 
will be referred to as the discussion proceeds. Regression 
results will be presented in each section that follows. 

1. Random Walk Model Bias (BIASRW) 

Table 9 provides the results of the multiple 
regression analysis for BIASRW. The results indicate that 
approximately 33% of the variation in random walk model bias 
was explained by the nine independent variables. However, 
only two variables — LRATE and FUTUPROD — were significant in 
explaining variations in the bias of random walk model cost 
predictions. The relationships between random walk model bias 
and each of these variables are depicted in Figure 1. 
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